Skip to content

GH-3708: Inline parquet.thrift - #3709

Open
divjotarora wants to merge 1 commit into
apache:masterfrom
divjotarora:inline-thrift
Open

GH-3708: Inline parquet.thrift#3709
divjotarora wants to merge 1 commit into
apache:masterfrom
divjotarora:inline-thrift

Conversation

@divjotarora

@divjotarora divjotarora commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

It is currently not possible to build working POCs for unreleased parquet-format changes in parquet-java because the parquet-format dependency must be updated using a released version. This blocks reference implementations from merging and will become a bigger issue if the current versioning proposal goes through and we start having more "preview" features where writes are expected to support unreleased spec changes behind feature flags.

What changes are included in this PR?

This PR removes the dependency on github.com/apache/parquet-format and instead adds an inlined copy of parquet.thrift (parquet-format-structures/src/main/thrift/parquet.thrift) as well as a "sidecar" metadata file (parquet-format-structures/src/main/thrift/parquet-format.version) to indicate the parquet-format version that's being inlined. This can be either an X.Y.Z semantic version or a commit hash.

There are two scripts added to help manage the vendored copy:

  1. dev/update-parquet-thrift.sh [$SHA | --version X.Y.Z updates the inlined parquet.thrift to match an upstream copy of parquet-format given either a commit hash (for POC development) or a version (for releases)
  2. dev/check-parquet-thrift-release.sh checks that inlined parquet.thrift mirrors a parquet-format release, not an unreleased commit. This is meant to be used in parquet-java releases only, not to block features from merging.

Are these changes tested?

Edit inlined parquet.thrift with a meaningless change and then validate it gets overwritten:

-- Validate diff
> git diff
--- a/parquet-format-structures/src/main/thrift/parquet.thrift
+++ b/parquet-format-structures/src/main/thrift/parquet.thrift
...

-- check-parquet-thrift-release should fail due to diff
> ./dev/check-parquet-thrift-release.sh
ERROR: inlined parquet.thrift differs from parquet-format 2.13.0.

-- Force an update
> ./dev/update-parquet-thrift.sh --version 2.13.0
Update complete:
  version:    2.13.0 (release sync)

-- check-parquet-thrift-release should now succeed 
> ./dev/check-parquet-thrift-release.sh
OK: inlined parquet.thrift matches parquet-format 2.13.0.

Also test updating to a commit rather than a version:

> ./dev/update-parquet-thrift.sh 2076361bb64e2de9ca6a8d06eda025a6fa4e9df6
Update complete:
  old commit: c47e2a66e88943fc46fde1b028a9432f14fdf5c0
  new commit: 2076361bb64e2de9ca6a8d06eda025a6fa4e9df6
  version:    UNRELEASED (POC sync — run --version <ver> for a release sync)

-- Validate it changed
> git diff --stat
parquet-format-structures/src/main/thrift/parquet-format.version |  4 ++--
parquet-format-structures/src/main/thrift/parquet.thrift         | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++--------

-- check-parquet-thrift-release should fail due to unreleased version
> ./dev/check-parquet-thrift-release.sh
ERROR: inlined parquet.thrift is not from a released parquet-format version.

Are there any user-facing changes?

No

Closes #3708

@divjotarora
divjotarora marked this pull request as ready for review August 5, 2026 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inline parquet.thrift instead of dynamically pulling in parquet-format

1 participant